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Abstract 


Like numerous others before us, we argue that the educational research community would be 
well served by a mutually created cyberinfrastructure that encourages and supports engagement 
by multiple design-based researchers in working toward answers to important theory-driven 
research questions, moving our field toward a “bigger science” approach. We reach out to 
members of the educational research community to join with members of other relevant fields, 
such as computer science, to design, build, support and participate in a mutually established 
cyberinfrastructure and user community to support design-based research. We seed the 


discussion by offering a concrete initial proposal to illustrate the possibility of implementing this 
idea. 


ii 


A Cyberinfrastructure for Design-Based Research: 


A Collaborative Design Project Proposal 
Sharon J. Derry, Alan Hackbarth, and Sadhana Puntambekar 


Design-based research (DBR) refers to a mixed-methods approach that involves the iterative 
and systematic design, development, and study of theoretically guided educational innovations in 
their implementation contexts (Barab & Squire, 2004; Collins, Joseph, & Bielaczyc, 2004; The 
Design-Based Research Collective, 2003; Reeves, Herrington, & Oliver, 2005). Although DBR 
is in early stages of development as a methodology and paradigm, its popularity among 
researchers, funding agencies, and journals is growing and spawning lively academic debates 
(e.g., Anderson & Shattuck. 2012; e.g., Dede, 2004; McKenney & Reeves, 2013). Ina 
penetrating commentary on the special issues devoted to DBR in Educational Researcher (2003) 
and Journal of Learning Sciences (2004), Dede (2004) worried that discussions had pushed the 
boundaries of DBR too far, rendering a sort of “Swiss Army Knife” for scholars trying to find a 
tool for too many purposes. He encouraged instead a bounded conceptualization (e.g., Collins, 
Joseph, & Bielaczyc, 2004), giving focus to the kinds of questions DBR studies can reasonably 
address: When is an initial implementation of an educational innovation successful enough to 
merit further investment in perfecting the innovation? When is it successful enough to warrant 
implementation and testing on a large scale? What generalizable knowledge about the conditions 
and reasons for success can be gleaned from a DBR study, and how can we use this knowledge 
to support effective translation of a successful innovation into a range of new contexts? Our 
comments that follow pertain mostly to these last two questions regarding how the DBR research 
enterprise might be improved and leveraged to help discover generalizable scientific knowledge 
needed to support full-scale implementation of successful educational innovations. 


Whether general scientific knowledge can be harvested from DBR is, however, a subject for 
debate. DBR crafts innovations by working within educational contexts that represent a complex 
mesh of goals, content, technology use, principles of learning, teaching methods, and 
accountability systems that must be woven together and shaped to a particular environment. 
Because any measure of success reflects the whole of the interacting variables that comprise the 
complex system, it is very difficult to isolate stable causal mechanisms within it or to translate 
lessons learned into other grade levels, students, schools, subject matter, types of technologies, 
etc. (Fadel & Lemke, 2006). For this reason, Anderson and Shattuck (2012) speculate that DBR 
may be more suitable for making and sustaining improvements in small-scale systems rather than 
contributing to large-scale and far-reaching systemic reform. 


Yet DBR is increasingly called upon to fill an important niche in the array of methods now 
used to improve educational practice and create generalizable principles about it (Collins et al., 
2004). For example, the Institute for Educational Studies (IES) currently funds ideas for 
educational innovation at several stages of increasingly larger funding, with DBR often the 
preferred approach for early and middle phases of an innovation’s history. DBR in these phases 
helps perfect and test a theoretically designed innovation, providing evidence pertaining to its 
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local success as well as implementation experience to support an informed translation, if 
watranted, into increasingly larger-scale contexts. Eventually, randomized experimental trials 
may produce generalizable judgments about an innovation’s effectiveness and, consequently, 
about the theory of learning upon which the implementation project was based. In this model, 
DBR, hypothetically, contributes to theory from a pragmatist’s perspective: Theories about 
learning in context are proposed and evolve through their various instantiations within a DBR 
research program. Experimental trials then produce confirmatory evidence, offering 
generalizable judgments about an innovation’s effect as well as evidence supporting the 
pragmatic value of the situated learning theory upon which the implementation was based. 


However, this passage to educational research Nirvana through DBR is a slow, methodic, 
expensive pilgrimage and, as recent discussions of DBR acknowledge, fraught with challenges. 
In fact, reviewers of DBR progress have been hard-pressed to identify any such completed 
journeys. Before considering those challenges and how to address them, it may be worth a brief 
digression to consider an alternative case, that is, the result of a lengthy educational research 
enterprise that did not build on a meticulous build-up of theory and implementation knowledge 
that DBR research strives to achieve. Consider, for example, the hundreds of studies of the 
effects of technology that have been conducted across a wide variety of educational settings over 
the last several decades. Many studies have been aggregated in several influential reports—e.g., 
ACT Policy Report: Evaluating the Effectiveness of Technology in Our Schools (2004), Cisco 
Systems/Metiri Group Technology in Schools: What the Research Says (2006), Milken 
Exchange on Education Technology (1999), and the National Middle School Association 
Research Summary: Technology and Learning (2007). These reports consistently acknowledge 
that using technology provides a small but significant increase in learning across all uses in all 
content areas (Fadel & Lemke, 2006; Schacter, 1999; Noeth & Volkov, 2004), but they offer 
little guidance for implementation and include substantial caveats. Has a torrent of research on 
educational technology produced only a trickle of knowledge? 


One problem relates to the inadequacy of the research methodologies employed to reach 
conclusions. “Most studies on the effect of educational technology on learning are correlational 
studies,” wrote Fadel and Lemke (2006). “Although such studies suggest what is working, they 
do not control for confounds that may provide alternate explanations for results.” Also, perhaps 
we have been mired too long in fruitless atheoretical research that engages experimental and 
statistical control to focus on effects of single features, such as “uses technology,” that are 
largely meaningless when considered separately from the complex, interacting factors 
constituting an entire ecology of teaching and learning. “Results and conclusions must be 
considered in the context of the interdependent set of variables in which the use of technology is 
embedded” (Fadel & Lemke, 2006). The unfolding DBR narrative is largely a research 
community’s response to this realization, and the story of its aspiration to carry out credible, 
theoretical scientific research of pragmatic significance in complex natural settings. 
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The Promise and Problems of Design-Based Research 


Unlike the experimental psychologist, who controls all aspects of an environmental recipe 
including the specific added ingredients associated with causal hypotheses about cognitive 
mechanisms, the DBR researcher eschews introducing artificial controls into educational 
environments. What DBR teams do is infuse an established educational ecology with new ideas 
and creations (an “innovation”), having in mind some theoretically derived hypotheses about 
how such infusions will alter the landscape to effect positive learning outcomes. Then begins the 
business of systematically studying that dynamic landscape in action, perhaps before but 
certainly during and after the innovation is introduced, searching for clues that will lead to an 
understanding of how the innovation and the ecology coalesce to effect learning. Researchers 
crisscross the landscape to collect data for analyses based on their hypotheses and a priori 
research questions, but they also attempt to capture other interactions that emerge as interesting, 
to increase the likelihood of fruitful unanticipated discoveries (e.g., Derry et al., 2010). In this 
effort to leave no potentially important stone unturned, DBR researchers typically document an 
innovations’ development and implementation history with detailed descriptions of situational 
contexts, rationales for designs and design changes at different phases, and learning and process 
outcomes. The overwhelming result is that every DBR project produces a substantial data 
collection. A widespread criticism and controversial issue dogging DBR is that it typically 
generates volumes of data that are never used. 


One reason for this current state of affairs is that there are no agreed-upon standards, goals, 
or structures to guide DBR scholars on what data to collect and archive. In addition to being 
unwieldy in size, DBR datasets are shaped in content and structure by whatever unique research 
questions and systems for data collection and archiving that an individual project or research 
group devises. Uniquely structured, unpublished datasets cannot be accessed or utilized, without 
great difficulty, by researchers and educators from outside the project. It is even less likely that 
such datasets will be combined and aggregated for analytic purposes. In fact, the DBR research 
enterprise, in all respects ranging from the formulation of research questions to the archiving of 
data, is so fragmented that sharing and advanced data mining, which could address important 
theory-based questions across many projects, are difficult to impossible. 


In discussing both promises and threats to the impact of “design-based implementation 
research,” Penuel, Fishman, Cheng and Sabelli (2011) argued that it is time to develop DBR as a 
more systematic form of inquiry and practice. They believe the DBR community must develop 
better norms and practices for theory development and for specification and testing of claims. 
They also called for developing standards regarding how evidence is used to guide design 
refinements, and practices for incorporating into studies multiple points of view and conflicting 
interpretations of data. Finally, they suggested standardized use of design rationales in the 
manner that professionals in fields such as architecture, urban planning, and software 
engineering articulate such rationales to clarify the purposes and history of the design process, 
and to help them reflect on and modify designs. Design rationales could serve to make public the 
ways that educational teams employ evidence to resolve conflicts, weigh competing approaches 
to improvement, and identify new areas of focus for their work. 
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Penuel et al. (2011) echo similar appeals by other analysts of DBR research. Collins, Joseph 
and Bielaczyc (2004), as recently cited in Anderson & Shattuck (2012), previously advanced the 
idea of a DBR infrastructure to support data archiving, sharing and collaboration. Noting that 
design experiments produce large amounts of data that go unanalyzed, they called for an 
infrastructure to allow researchers from outside the original design team to access and analyze 
the data collected in large studies. Not surprisingly, Anderson and Shattuck’s recent review 
(2012) failed to uncover any evidence of such sharing currently taking place. Such sharing would 
require the DBR community to have standard protocols and systems for archiving of data in a 
format that supports multiple forms and levels of access, sharing and analysis, while adequately 
protecting the privacy and identities of human subjects. 


It seems, therefore, that the educational research community would be well served by 
mutually creating an infrastructure that would encourage and support collective engagement by 
multiple design-based researchers in working toward answers to important theory-driven 
research questions, moving our field toward a “bigger science” approach. It is this spirit that we 
reach out to members of the educational research community to join together with members of 
other relevant fields, such as computer science, to design, build, support and participate in a 
mutually established cyberinfrastructure and user community for DBR. In 2004 Chris Dede 
challenged the DBR community to engage in an evolutionary dialogue about the purpose and 
processes of DBR that would define DBR in a realistic way. He was not optimistic that the 
community would be able to do so at that time. Perhaps the time is now. 


The Workflow Visualization System (WVS): A Design Proposal 


We seed the discussion by offering a concrete initial proposal for designing a 
cyberinfrastructure. Based on the dissertation research of Alan Hackbarth (Hackbarth, 2011), this 
proposal is offered as a feasibility demonstration and conversation starter rather than a final 
solution path. 


Conceptual Foundations 


DBR is conducted within the “blooming, buzzing confusion” of classrooms and other 
authentic learning environments (Brown, 1992, p. 141). Many variables are at play within design 
experiments, including some that are not anticipated and cannot be controlled (Collins, Joseph, & 
Bielaczyc, 2004); e.g., events such as the interrupted availability of a computer server or the 
timing of a spring break. Yet it is important when doing DBR to identify the critical variables of 
a design and how they fit together (Collins et al., 2004). In order to evaluate an implementation 
of an instructional design one needs to analyze each case in terms of its key elements and their 
interactions. Some elements will be implemented more or less as designers intended, some will 
be changed to fit circumstances, and some will not be implemented at all. A profile of how each 
critical element was implemented and how they worked together toward the designer’s goals is 
needed. Further, because each variable is part of a systemic whole it may be impossible to 
change one aspect of the system without creating perturbations in others (Brown, 1992). These 
perturbations and their effect on student learning need to be identified and accounted for in 
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iterative cycles of design, enactment, analysis, and redesign in authentic settings that constitute 
DBR (Barab & Squire, 2004; Collins, 1992). 


It is important to document designs at a level of detail appropriate to the research questions 
and design goals of the experiment, and critical to record all major design changes at that level. 
Major changes mark the borders that differentiate one phase of an implementation from the next 
(Collins, 1992). A well-documented design history that includes well-archived, accessible 
student outcome data can provide evidence for causal impact on measured student outcomes. It 
facilitates theory building that impacts student learning and allows research audiences to evaluate 
the credibility of design decisions and the quality of lessons learned from the research (Barab & 
Squire, 2004). However, in the process of documenting and studying design, researchers collect 
large amounts of data throughout an intervention—e.g., video, student produced artifacts, and 
activity logs from online interactions—often more data than they have time or resources to 
analyze (Barron, 2007; Brown, 1992; Collins et al., 2004). To ensure that design researchers 
maximize their use of collected data to develop rich representations of critical variables and their 
interactions in a learning environment, analytical procedures that organize, document, and 
archive data in an easily accessible format that facilitates more efficient deeper analyses are 
needed (Merriam, 1988, p. 124). Such procedures also maximize potential for access and reuse 
of instructional designs, assessments, and learning outcome data by a broader research 
community. 


Intermediate Representations 


Barron (2007, p. 178; as cited in Derry et al., 2010) discussed the emergence and value of 
intermediate representations as a response to the need for better analytical procedures for 
complex field-based datasets. Intermediate representations that organize and display data are 
important for identifying what to analyze and for understanding patterns within and across 
segments of field data. Barron identified several examples of intermediate representations that 
are either created during data collection or are derived from an initial macro-analysis of field 
records. These include content logs, which are often created in the form of field notes that index 
data while it is being collected; table-based flow charts that chronologically catalog events of a 
group or individual learning experience and highlight significant events for deeper analysis (Ash, 
2007); descriptive diagrams of interactions that illustrate participants’ relations to one another 
and resources in an informal learning situation (Angelillo, Rogoff, & Chavaj, 2007); and 
conversation maps that show the flow of discourse in a learning situation (Barron, 2003). 
Intermediate representations provide organizing structures that help research teams develop a 
sense of the corpus of data and facilitate the selection of episodes for further detailed analyses 
(Barron, 2007, p. 179). 


Macro-level intermediate representations. Macro-level intermediate representations satisfy 
a core characteristic of DBR: that the goal of developing sharable, adaptable learning 
interventions with practitioners is intertwined with the goal of developing theories of learning 
(Collins, 1992, The Design-Based Research Collective, 2003). Standardized systems of 
intermediate representation could facilitate DBR by providing users with a means of representing 
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both major elements that frequently appear in designs across contexts, as well as their major 
interrelationships, in much the same way this has been accomplished in physical sciences (De 
Roure, Goble, & Stevens, 2007). Wide adoption of an adaptable but standardized 
representational system would allow researchers to hypothesize about, even conduct virtual 
experiments on, the learning effects of including, rearranging, or redefining the contents of 
elements across contexts. 


Visualization. In writing about information processing in the world of business, David 
Tegarden (1999) observed that decision makers suffered from information overload while at the 
same time underutilizing large amounts of data. Because the majority of the brain’s activity that 
deals with processing sensory data deals with analyzing visual images, he argued, visualization 
technologies help resolve this dilemma. Humans naturally look for structure, features, patterns, 
trends, anomalies, and relationships in data (Grinstein & Ward, 2002). A visualization harnesses 
the perceptual capabilities of the human visual system and allows a viewer to (a) examine a large 
amount of data, (b) keep an overview of the whole while pursuing details, (c) keep track of many 
things by using the display as an external working memory, and (d) produce an abstract 
representation of a situation through the omission and recoding of information (Card, 2008). 
Larkin and Simon (1987) argued that diagrams can be superior to written representations because 
they can (a) group related information together, (b) use location to aid in information search, (c) 
aid in many perceptual inferences, and (d) support efficient computational processes. 
Visualizations amplify cognition by reducing the search for information, enhancing the detection 
of patterns, enabling perceptual inference operations, using perceptual attention mechanisms for 
monitoring, and by encoding information in a manipulable medium (Card, 2008). Tegarden 
(1999) cautions that the purpose of visualization is not to replace empirical analysis, but instead 
to focus such analysis by exploiting the human visual system to extract information from data 
that allows decision makers to determine areas of interest or promise where further exploration 
should be done. 


A Visual Workflow System (VWS) for DBR 


One feasible approach is to construct a system that uses visual workflow to construct 
intermediate representations that help structure research and sharing of research about 
instructional interventions (Hackbarth et al., 2010). This approach has provided a valuable 
support for our own DBR, although because we lack the technological tools that automate our 
approach, our current method is time consuming. It is also limited in that there is no automated 
mechanism that allows researchers to easily annotate elements of an intervention; for example, 
noting the rationale for the inclusion of an instructional resource or adding a reflective note about 
an idea for lesson modification. However, a fully developed VWS would add these 
functionalities, making it efficient for helping DBR researchers archive, document, and explore 
data collected from interventions. It would also make the data easily available and browsable by 
other educators and researchers. 


Widespread adoption and use of such a system is plausible because many learning 
environments share key structural similarities. They (a) set learning goals and coordinate a series 
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of activities to accomplish those goals, (b) design or identify ways to collect data on student 
learning to assess accomplishment of goals, (c) relate resources (e.g., readings, demonstrations, 
video) to learning activities, (d) organize workspaces, and (e) scaffold student interactions during 
activities. Based on feedback from case studies conducted with research partners at Rutgers 
University (Hackbarth, 2011), we successfully applied a workflow metaphor to create 
understandable, sharable, multilevel standardized intermediate representations of educational 
interventions that show (a) the essential flow of lesson tasks, (b) the relation of resources to 
tasks, (c) the outcomes of tasks, including evidence of student learning and performance, and (d) 
details on how learners are expected to and/or actually do interact with lesson resources and one 
another. 


The Workflow Management Coalition defines workflow in a business environment as “[t]he 
automation of a business process, in whole or part, during which documents, information, or 
tasks are passed from one participant to another for action, according to a set of procedural rules” 
(Hollingsworth, 1995). Broadly stated, a workflow “participant” (i.e., human being or 
technological tool) receives data in some form as input, acts upon it with a process that may or 
may not be fully scripted in advance, and its output may become or inform input for the next 
“participant.” Although contexts and interactions in the world of business are generally more 
controlled and predictable than in education, a business processes metaphor has utility in 
describing the design and implementation of an educational intervention: instruction can be 
generally described as a series of processes where documents, information, or learning tasks are 
passed to learners with the intent that they will be processed by some action and (measurable) 
learning outcomes will be produced. 


In applied science, workflows are not simply digital data objects, they capture pieces of 
scientific process—they are valuable knowledge assets in their own right because they are 
graphical representations of “know-how” that is often tacit. The idea of reuse (De Roure et al., 
2008) suggests that there may be a standardized way of doing particular processes where the 
inputs change but the process, or the rationale for the process, remains consistent across 
implementations. Reuse can occur effectively at multiple levels: a scientist can reuse a workflow 
with different parameters and data, fragments and patterns of a workflow can be reused to 
support science outside their initial application, or they can provide a means of codifying, 
sharing, and spreading the workflow designer’s practice. Parallels can be drawn to teachers or 
designers who introduce new inputs (e.g., data or concepts from instructional materials) while 
reusing a pedagogical process (e.g., a problem-based learning activity) from lesson to lesson, and 
to share adaptable interventions. 


Inspired by increasing use of workflow languages to capture, document and control processes 
in both applied science research and business applications (Ailamaki, Ionnidis, & Livny, 1998; 
Cardoso, Bostrom, & Sheth, 2004; Ludascher et al., 2005; Mentzas, Halaris, & Kavadias, 2001), 
and by trends toward reuse and recombination based on standard workflows and shared 
databases in the physical sciences (De Roure, Goble, & Stevens, 2007; Gil et al., 2007), we 
explored the potential of workflow visualization to help document instructional activities offered 
in an online learning environment for the purposes of (a) supporting DBR related to incremental 
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development of the activities and related theories that propose causal connections between 
design features and student learning outcomes; and (b) sharing adaptable, tested workflows of 
enacted educational interventions with other researchers and teachers. 


DBR itself can be described hierarchically as a workflow, as summarized in Figure 1. 


Figure 1. Steps in a design-based research workflow 


Step 1 Step 2 Step 3 Step 4 
Develop Testable Theoretical Implement Designed Systematically Analyze Data Evolve Design | 


Rationale to Guide Design for Intervention in on Both Design and Theory | | 
Desired Learning Goals Authentic Context Implementation and Learning : 


The implementation of a designed intervention at, for example, Step 2—a thoughtfully 
orchestrated collection of learning activities—can be unpacked and represented as a workflow 
visualization having varied levels of detail. At an intermediate level that we call the macro- 
workflow or lesson level, workflow visualizations facilitate (a) identifying and organizing 
(thereby helping to gain control over) many types of variables and learning outcome data that 
intersect during an educational intervention, and (b) sharing adaptable educational interventions 
that produce known outcomes with other educators who can modify the general framework 
represented by the workflow expression to accommodate local contexts. Additional levels might 
provide progressively finer-grained representations that both constitute and facilitate deeper 
analysis (Step 3) in the iterative process of DBR that may include both hypothesis-driven and 
exploratory studies of learning. 


Workflows feature visual representations of data. Edward Tufte (1990. p. 33) notes: 


We envision information in order to reason about, communicate, document, and preserve 
[that] knowledge—activities nearly always carried out on two-dimensional paper and 
computer screens. Escaping this flatland and enriching the density of data displays are the 
essential tasks of information design. Such escapes grow more difficult as ties of data to 
our familiar three-space world weaken (with more abstract measures) and as the number 
of dimensions increases (with more complex data.) 


Visually representing the density, complexity, and dimensionality of a learning environment 
within a limited n-dimensional space requires compromise; some elements of a designed 
intervention need to be privileged in their representation, yet technologies must allow access to 
representations of all the critical variables and reveal the logic of their interrelations. Tufte’s 
writings (1990, 1997) discuss effects and challenges of several such techniques for “enriching 
the density of data displays”: micro/macro renderings, in which exquisite detail leads to both 
focused micro-readings of individual elements of the design and cumulates into larger coherent 
structures; using layering and separation to reduce noise and distraction and create relative strata 
amongst related variables; using small multiples for comparisons; using color to encode abstract 
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information; and narratives of time and space that account for the temporal and spatial relations 
among and among elements in a system. 


Tufte’s ideas and those of information visualization researchers such as Card (2008), Spence 
(2001), and Ware (2008) inform a rationale that can be consistently applied across different 
educational environments for the visual display of instructional workflows, and that takes into 
account the affordances of the tools used to create the representations—e.g., HTML documents 
can be hierarchically “layered” using hyperlinks or encoded using cascading style sheets. 


A Visual Workflow System for DBR: What Elements are Needed? 


What elements (or variables) should be described and included in a “standardized” system of 
workflow visualization for designed educational interventions? Some probable ones include, as 
examples: students, instructors, small groups, readings, video, worked examples, discourse, 
assessment, rubrics, presentation, and analysis. There are many possibilities, and their 
interrelationships within different designs are complex and multidimensional. Fortunately two 
sources in the DBR and workflow literatures provide starting points, enabling us to explore these 
questions systematically. Bielaczyc’s (2006) Social Infrastructure Framework highlights four 
dimensions that researchers need to consider as they design learning environments: cultural 
beliefs; practices; socio-techno-spatial relations; and interaction with the outside world. The 
Workflow Management Coalition (Hollingsworth, 1995) identifies five key perspectives—what, 
when, by whom, using what data and what tools—that need to be included in a valid workflow 
expression. These perspectives seem to have significant overlap and are well aligned with other 
important theoretical frameworks frequently employed in the learning sciences, such as Activity 
Theory (Engestrém, 1999; Kaptelinen & Nardi, 2009; Kuuti, 1996). 


Beginning with these frameworks, we conducted research leading to a unique system of 
visual display to support (a) cross-site sharing, with researchers and teachers, of adaptable 
instructional interventions invented at various sites, and (b) DBR by allowing researchers to 
visually depict designed instructional interventions and their assessment output at varying levels 
of detail, which facilitates data organization, data sharing, and achievement of experimental 
control. The design of this system itself followed a DBR paradigm. Our primary research 
questions were: (a) Can we invent a standard workflow-based methodology that can detect and 
document differences in designed interventions, solve problems of control and efficiency in 
DBR, and inform theory-based design changes that improve student outcomes in the context of a 
particular online course that serves as a case study? (b) Can we develop a sound argument that 
the methods developed for this case study are more widely applicable to the work in 
environments beyond? 


Context of the Study and Data Sources 


The context for this study was a special hybrid section of Human Abilities and Learning 
(HAL Online), an innovative experimental section of an educational psychology course taught 
regularly at the University of Wisconsin—Madison. The study was conducted in two iterations of 
the course, spring 2010 and fall 2010, within a 4-week unit on children’s thinking that focused 
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on mathematical development. This reusable unit includes four 1-week online sessions and one 
face-to-face class meeting. The activities that students engage in online include: (a) completing 
assigned readings and exploring topic-related web resources, (b) viewing videos of children’s 
problem solving, (c) participating in a small group discussion to compare video cases and 
analyze teaching, learning, or curricula, argue different positions and use course ideas to support 
arguments, and (d) taking an essay quiz and completing a reflective blog at the end of a unit. In 
the face-to-face meeting students worked collaboratively in small groups to inductively solve 
mathematics problems using Unifix Cubes, then presented and discussed their solutions and 
experiences in terms of the course material. The mathematical understandings acquired in these 
sessions deepened and supported further online collaborative analysis of videos. 


Steps of the investigation: 


1. Mining data from course modules and activity logs of online interactions, and from video 
and interview data collected during the face-to-face meeting, for the purpose of creating 
initial macro-workflow expressions representing the instructional unit. 


2. Interviewing project researchers and teachers from multiple institutions to obtain 
reactions to initial workflow expressions and the proposed elements used to create them. 


3. Generating a pallet of reusable workflow icons from which workflow expressions could 
be generated, taking into consideration feedback from project members. 


4. Iteratively refining the workflow visualization system based on case studies from 
discussions with project researchers, project managers, teachers, and software analysts 
and developers given tasks requiring them to use workflow elements to create workflows 
(Hackbarth, 2011). 


The resulting methodological tool represented a synthesis of ideas about social learning 
environments and their representation (Angelillo, Rogoff, & Chavajay, 2007; Ash, 2007; Barron, 
2007; Bielaczyc, 2006) from Learning Sciences literature, the Information Control Nets 
workflow modeling language (Rembert, 2006) with theoretical grounding in data mining 
(Fayyad, Piatetsky-Shapiro, & Smyth, 1996; van der Aalst, Weijters, & Maruster, 2004), 
workflow management (Hollingsworth, 1995; Jablonski & Bussler, 1996), and information 
visualization (Card, 2008; Spence, 2001; Tufte, 1990, 1997; Ware, 2008) . The major features of 
this tool are described below and can be experienced dynamically at our website: 
http://vmc.wceruw.org/workflow/workflow.html 


Features of the WVS 


We designed icons to build the workflow visualizations and provide visual cues about tasks, 
information inputs, data outputs (e.g., student learning outcomes), and other lesson elements 
such as descriptions of the workspace and learning assessments. The various icons can be seen at 
different levels of Figure 2. Lesson (II) and Lesson Component (III) levels have task icons—they 
are circular and contain visual cues about the nature of the task and the participation structures 
(note: each workflow comes with a legend of icon descriptions). Level III, an expansion of the 
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small group activity at Level II, illustrates two individual student tasks—recalling prior 
knowledge and viewing a video—and a small group discussion. Information visualization 
principles involving color and shape are utilized to distinguish input (blue, square, arrow-in) and 
output (red, square, arrow-out) icons, which serve as hyperlinks that, when clicked on or hovered 
over reveal more information about a resource, or data visualizations. Finally, hexagon-shaped 
lesson element icons can be seen at Levels II and III; these include such things as description of 
the workspace (the small icon at the far-left at Level II), lesson goals, assessments, and task 
instructions (Level III). Lesson assessment icons are expandable to show another set of 
trapezoid-shaped icons for formative assessments used for a lesson, scoring rubrics, and 
summative student performance data. Although not shown in Figure 2, clicking on the Unit 
Assessment icon at Level I will also provide links to unit-level assessments and student 
performance data. 


Figure 2 shows the hierarchical and networked organization of a multi-level workflow 
expression. A dynamic version, which initially displays only the first two levels and opens other 
levels based on how the user moves the cursor or clicks the mouse, can be accessed at 
http://vmc.wceruw.org/workflow/ebd-workflow.html. Figure 2 illustrates the depth of information 
that a workflow representation contains. The unit workflow (Level I) places a lesson within the 
context of a unit and serves as a navigation tool; clicking on a lesson’s icon opens that lesson’s 
workflow. The centerpiece of the representation is the lesson, or macro-workflow (Level II). 
Task icons are arranged in the order in which they are designed (or recommended) to be 
accomplished. Dashed lines in the workflow indicate that, while the tasks are given to the learner 
in a specified order, the learner might start from another point in the lesson, or revisit tasks. 


As a user moves the cursor over each task icon, a data repository (i.e., a rectangular box that 
may hold text only, text and input/output icon(s), or graphical data) opens. The display may 
include hyperlinks to actual resources used in the lesson or data generated by students doing a 
task. If moving the cursor over a task icon does not trigger a box to open, then clicking on the 
icon will open a sub-workflow in a new window (right-hand side of Level III). A sub-workflow 
organizes and displays information about that collection of tasks and functions in a way similar 
to a macro-workflow; sub-workflow tasks may produce multiple outputs. In Figure 2, the 
discussion task shows links to compliance data (how and how often a student interacts in the 
discussion), a transcript, and an analysis of group interaction (a micro-workflow). 


Functions of the WVS. Primary functions of a workflow representation are to organize the 
information and (as they become available) research analyses based on learning outcomes data 
associated with an intervention, and to support archiving, retrieving, and ethical (with respect to 
student privacy) sharing of design history, outcome data, and analytics associated with DBR. We 
anticipate that a researcher or educator will have one of two potentially overlapping reasons for 
viewing and interacting with the representation. One involves reuse, possibly after some type of 
a modification, in which a lesson is borrowed by someone with similar goals or taught iteratively 
by the designer. A workflow visualization preserves the details of a lesson and assessed learning 
outcomes, thereby making it available for sharing, reflection, or adaptation. Second, a system’s 
users may be doing research, in which case the workflow representation preserves all data in an 
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organized fashion. The visual nature of the representation allows researchers to (a) quickly trace 
the relationships of different elements of an instructional design, which affords comparison and 
control across iterations, and (b) potentially see patterns that lead to useful conjectures and 
suggest avenues for further analyses. 


In sum, building on theories from the learning sciences and other related literatures regarding 
what features of learning environments are important to model, our proof-of-concept prototype 
served to demonstrate how a Moodle-based system for online course management might 
simultaneously serve as the basis for a cyberinfrastructure to support DBR researchers in 
conducting iterative, theory-based online course development, and in facilitating their data 
collection, archiving, sharing, and mining across projects of various sizes and types. We believe 
the concept is sufficiently flexible such that, with appropriate enhancements, it could support 
researchers using varied analytic approaches and conducting research in varied environments. 
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Figure 2. An exploded view of the multiple levels of information and data included in a workflow visualization 
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Moving Ahead 


Next Steps 


Can the complex richness of DBR be organized and tamed to enable more systematic, 
insightful, efficient, community-based inquiry? We propose to address this question by 
organizing a research community around a project to design, build, and test a prototype 
cyberinfrastructure for supporting DBR for educational improvement. The cyberinfrastructure 
we envision will promote systematic collection, archiving, sharing, and collaborative analysis of 
data from many DBR studies. We envision a standardized yet flexible web-based system that 
will help researchers document and archive the development and implementation history of 
designed educational interventions, including their contexts, rationale, iterations, and outcomes 
across a wide range of projects. 


Our next step is to assemble a team and embark on a 3-year NSF-supported collaborative 
design project that will synthesize recent work in DBR methodology (e.g., Bielaczyc, 2013) with 
relevant work in areas such as information visualization, machine learning and analysis, learning 
analytics, and online data mining to create a system with the following kinds of functionalities, 
although this is not a final or complete description: 


e document and share learning-environment designs, including inputs such as activity 
structures, contextual information, curricular and assessment materials, learning 
technologies, theoretical design rationales, hypotheses, and outcomes of design 
experiments; 


e provide a standardized web-based resource that will support researchers in designing 
instructional studies and ethically capturing, organizing and archiving large quantities of 
data from disparate sources (e.g., instructional materials, assessment, video data) that will 
be generated when implementing multiple design iterations; 


e provide the ability to capture and document design changes, both deliberate design 
iterations made by researchers or those that occur as products of “on-the-fly” 
instructional decisions or unforeseen circumstances; 


e present real-time, multi-level visualizations that will allow large datasets to be exhibited, 
shared, traversed, explored, and viewed at different grain sizes and levels of detail to 
facilitate archiving, sharing, and collaborative data analysis; and, 


e provide some level of automatic analysis of data and data mining to facilitate and speed 
the analytic process and promote the pursuit of some agreed-upon research questions of 
central importance to the field with data mined from across projects. 


Imagine the Future 


Data mining. The vision for data mining and instructional analytics within a collective DBR 
cyberinfrastructure can be contrasted with less theoretical approaches such as those associated 
with MOOCS and other large courses supported by online management systems. These current 
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approaches generally assume that massive amounts of data produced by large-scale innovations 
can be processed with machine-learning algorithms to discern patterns that may represent 
important discoveries about how people learn. However, neither convenient datasets no matter 
how large, nor intelligent algorithms no matter how sophisticated, will likely yield fruitful results 
if they are barren with respect to current learning-science knowledge and theory. DBR takes a 
theory-guided perspective that involves design of learning environments to collect planned 
datasets that can be analyzed to address what researchers believe are scientifically important 
questions. That a standardized, community-built cyberinfrastructure can support systematic, 
theory-based collective mining of data across DBR projects represents a novel and possibly 
important approach to instructional analytics. 


Virtual experimentation within DBR cyberinfrastructure. This concept is intriguing 
because it suggests an alternative to randomized control trials (RCT) as a possible end game for 
successful innovations. RCT’s require that massive amounts of time and resources be invested in 
perfecting, scaling, and testing what amounts to single cases of educational innovation. Within 
this strategy, the complex, interacting environmental factors that influence teaching and learning 
in context are controlled through a combination of sophisticated statistical, sampling, and 
experimental procedures that require very large sample sizes and the assembling of massive 
cooperating teams. The sheer size of such projects requires that measures of success depend 
heavily on large-scale standardized accountability systems that may not be sensitive to critical 
learning outcomes deemed important for theory-driven research. Such projects generally require 
aggregating complex educational features into “intervention packages” that can be implemented, 
studied and compared across school districts. The requirement for treatment fidelity and control 
is at odds with the DBR notion of adapting interventions to meet the needs and increase their 
effectiveness within the implementation contexts in which they are studied. 


Contrast the value and cost of such efforts with the possibility of mining an already existing 
and continuously growing bank of quality-controlled DBR data, designed to increase the 
likelihood of containing what a qualified research community needs to address its collective 
questions, to compare, for example, key features of successful versus less successful 
implementation stories in schools with selected characteristics. 


Concluding Comment 


The project we propose might develop our model further by building a workflow 
visualization system that interfaces with an existing course-management system such as Moodle 
or Sakai and that automatically constructs workflow diagrams of lessons and displays (on 
request) data collected as learners engage in online learning activities. The system may serve as 
an interface to provide the same functionality for other learning environments, including hybrid 
or face-to-face settings where different information or communication technologies might be 
used or adapted to upload data about interventions and learner-produced artifacts that are created 
outside the course-management system. 


Or, this collaborative design project engaging an entire design-research community might 
take new and unanticipated directions. 
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